Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition

Existing Scene Text Recognition (STR) methods typically use a language model to optimize the joint probability of 1D character sequence predicted by visual recognition (VR) model, which ignore 2D spatial context semantics within and between instances, making them not generalize well arbitrary shape scene text. To address this issue, we make first attempt perform textual reasoning based on in paper. Technically, given segmentation maps VR construct subgraph for each instance, where nodes represent pixels it edges are added their similarity. Then, these subgraphs sequentially connected root merged into complete graph. Based graph, devise graph convolutional network (GTR) supervising with cross-entropy loss. GTR can be easily plugged representative STR models improve performance owing better reasoning. Specifically, our namely S-GTR, paralleling segmentation-based baseline, effectively exploit visual-linguistic complementarity via mutual learning. S-GTR sets new state-of-the-art six challenging benchmarks generalizes multi-linguistic datasets. Code is available at https://github.com/adeline-cs/GTR.